Papers with object detection
Modularized Zero-shot VQA with Pre-trained Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on zero-shot visual question answering does not explicitly consider multi-step reasoning chains, making them less interpretable compared with a decomposition-based approach. |
| Approach: | They propose a modularized zero-shot network that explicitly decomposes questions into sub reasoning steps and is highly interpretable. |
| Outcome: | The proposed model decomposes questions into sub reasoning steps and is highly interpretable. |
GRACE: Gradient Harmonized and Cascaded Labeling for Aspect-based Sentiment Analysis (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies ignore aspect terms interaction when labeling polarities . aspect terms extraction and aspect sentiment classification are two fundamental tasks . |
| Approach: | They propose a GRadient hArmonized and CascadEd labeling model to solve the imbalance issue . they extend the gradient harmonized mechanism used in object detection to aspect-based sentiment analysis . |
| Outcome: | The proposed model achieves consistency improvement on multiple benchmark datasets and generates state-of-the-art results. |
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing vision-language pre-training methods use a two-step training procedure to learn visual features from image-text pairs. |
| Approach: | They propose a vision-language pre-trained model for V+L understanding and generation using a unified Transformer framework. |
| Outcome: | The proposed model can learn visual representation and semantic alignments between image and text on visual-text pairs and on visual processing tasks. |
LangNav: Language as a Perceptual Representation for Navigation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language navigation use visual features as the perceptual representation of a visual representation of an agent's egocentric panoramic view. |
| Approach: | They propose to use off-the-shelf vision systems to convert an agent’s egocentric panoramic view into natural language descriptions. |
| Outcome: | The proposed approach improves on the R2R VLN benchmark by using synthetic trajectories from a prompted language model and domain transfer where a policy learned on one simulated environment (ALFRED) is transferred to another (more realistic) environment and combining both vision- and language-based representations. |
VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods to “vet” labels from noisy captions for weakly-supervised object detection are limited for object detection. |
| Approach: | They propose a technique to “vet” labels extracted from noisy captions and use them for weakly-supervised object detection without any bounding boxes. |
| Outcome: | The proposed method improves WSOD without label vetting by 30% on five datasets. |
Domain-Specific Lexical Grounding in Noisy Visual-Textual Documents (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing image-text grounding approaches require detailed annotations, authors say . existing methods are difficult to adapt to unlabeled multi-image, multi-sentence documents, they say . |
| Approach: | They propose a method that can learn contextual meanings from unlabeled documents . they demonstrate that a simple unsupervised clustering-based method can be useful . |
| Outcome: | The proposed method is particularly effective for local contextual meanings of a word . existing image-text grounding methods are difficult to adapt to unlabeled multi-image, multi-sentence documents . |
Multilingual Image Corpus – Towards a Multimodal and Multilingual Dataset (2022.lrec-1)
Copied to clipboard
| Challenge: | The goal of the project Multilingual Image Corpus is to provide a large image dataset with annotated objects and object descriptions in 24 languages. |
| Approach: | They propose to provide a large image dataset with annotated objects and object descriptions in 24 languages. |
| Outcome: | The project provides a large image dataset with annotated objects and object descriptions in 24 languages. |
Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. |
| Approach: | They propose two approaches to contextualise visual entities in a multimodal setup by using verbalised scene graphs and masked relation prediction. |
| Outcome: | The proposed models can learn better representations from weakly-supervised relations data. |
See It All: Contextualized Late Aggregation for 3D Dense Captioning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent approaches to 3D dense captioning struggle with contradicting objectives . SIA generates captions with different region of interest and aggregates them afterwards . |
| Approach: | They propose a transformer pipeline that engages in 3D dense captioning with a new paradigm . SIA decodes two sets of queries—context query and instance query—and then aggregates them afterwards . |
| Outcome: | The proposed pipeline generates captions with different region of interest and aggregates them afterwards to enhance local-global sensitivity. |
DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning (2026.eacl-long)
Copied to clipboard
Nithin Sivakumaran, Justin Chen, David Wan, Yue Zhang, Jaehong Yoon, Elias Stengel-Eskin, Mohit Bansal
| Challenge: | a key strength of human intelligence is the ability to debate and discuss reasoning with others. |
| Approach: | They propose a multi-agent framework that uses disagreements between visual agents to identify useful visual tools that can resolve inter-agency disagreement. |
| Outcome: | The proposed framework beats the strongest baseline on A-OKVQA and MMMU, respectively. |
R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding (2025.findings-acl)
Copied to clipboard
Joonhyung Park, Peng Tang, Sagnik Das, Srikar Appalaraju, Kunwar Yashraj Singh, R. Manmatha, Shabnam Ghadar
| Challenge: | Existing vision-only GUI agents ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. |
| Approach: | They propose a visual agent model for GUI automation that leverages zoomed-in region proposals for precise element localization. |
| Outcome: | The proposed approach improves state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. |
ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for acquiring local visual information are limited . existing methods for named entity recognition are redundant or insufficient . |
| Approach: | They propose an Entity Spans Position Visual Regions module to obtain visual regions corresponding to entities in the text. |
| Outcome: | The proposed method achieves the SOTA on Twitter-2017 and competitive results on Twitter 2015 . previous efforts have yielded promising results, but they still fall short in selecting visual information. |
What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks. |
| Approach: | They propose a Gaussian-Noise-free pipeline for mechanistic interpretability in VLMs that introduces Semantic Image Pairs corruption, the first visual counterpart to Symmetric Token Replacement for text. |
| Outcome: | The proposed pipeline identifies a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |
DetGPT: Detect What You Need via Reasoning (2023.emnlp-main)
Copied to clipboard
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, Tong Zhang
| Challenge: | Recent advances in the field of computer vision have enabled more effective and sophisticated interactions between humans and machines. |
| Approach: | They propose a reasoning-based object detection paradigm that leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user’s instructions and the visual scene. |
| Outcome: | The proposed method enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity. |
An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have shown impressive generalization ability on vision and language tasks, but their spatial understanding is under-explored. |
| Approach: | They construct a VQA dataset to analyze LMMs' spatial reasoning capabilities. |
| Outcome: | The proposed model is stronger at basic object detection than complex spatial reasoning. |
Grounded Semantic Role Labelling from Synthetic Multimodal Data for Situated Robot Commands (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing symbolic parsers lack flexibility to operate in complex, dynamic environments. |
| Approach: | They propose a framework that combines frame semantics with perceptual grounding to enable robots to interpret commands via multimodal logical forms. |
| Outcome: | The proposed framework produces over 11,000 image-command pairs and lowers the cost of manual parsers. |